Research on the classification of CT images of COVID-19 by pre-training model based on self-attention mechanism

Nowadays, with the outbreak of COVID-19, the prevention and treatment of COVID-19 has gradually become the focus of social disease prevention, and most patients are also more concerned about the symptoms. COVID-19 has symptoms similar to the common cold, and it cannot be diagnosed based on the symptoms shown by the patient, so it is necessary to observe medical images of the lungs to finally determine whether they are COVID-19 positive. As the number of patients with symptoms similar to pneumonia increases, more and more medical images of the lungs need to be generated. At the same time, the number of physicians at this stage is far from meeting the needs of patients, resulting in patients unable to detect and understand their own conditions in time. In this regard, we have performed image augmentation, data cleaning, and designed a deep learning classification network based on the data set of COVID-19 lung medical images. accurate classification judgment. The network can achieve 95.76% classification accuracy for this task through a new fine-tuning method and hyperparameter tuning we designed, which has higher accuracy and less training time than the classic convolutional neural network model. © 2023 SPIE.

Face mask detection based on improved YOLOv3 algorithm

In the current situation of COVID-19 prevention and control, wearing masks remains an important way to prevent the transmission of the Novel Coronavirus. Aiming at the problem that the detection accuracy of the traditional YOLOv3 algorithm can still be improved, this paper proposes an improved yolov3 algorithm and applies it to the practical problem of detecting whether to wear a mask. Firstly, the algorithm introduces the residual structure of structural reparameterization in the feature extraction network named Darknet53 of YOLOv3 to obtain the input features;Secondly, the SimSPPF (Simplified Spatial Pyramid Pooling-Fast) is introduced to enhance feature extraction;Finally, an improved attention mechanism is introduced to make the model focus on regions with more prominent features. Besides, in order to ensure the accuracy of target detection, CIoU and Focal loss function was used in the training process. The results show that compared with the traditional YOLOv3, the detection accuracy of the improved algorithm for normal face and mask face is improved by 16.98% and 7.30% respectively, and the mAP is improved by 12.14%, which can meet the requirements of daily use and lay a foundation for rapid face recognition when wearing mask. () © 2023 IEEE.

Mask Wear Detection Algorithm Based on YOLOv5m

Wearing a mask during the COVID-19 epidemic can effectively prevent the spread of the virus. In view of the problems of small target size, crowd blocking each other and dense arrangement of targets in crowded places, a target detection algorithm based on the improved YOLOv5m model is proposed to achieve efficient detection of whether a mask is worn or not. This paper introduces four attention mechanisms in the feature extraction network based on the YOLOv5m model to suppress irrelevant information, enhance the information representation of the feature map, and improve the detection capability of the model for small-scale targets. The experimental results showed that the introduction of the SE module increased the mAP value of the original network by 9.3 percentage points, the most significant increase among the four attention mechanisms. And then a dual-scale feature fusion network is used in the Neck layer, giving different weights to the feature layers to convey more effective feature information. In the image pre-processing, the Mosaic method was used for data enhancement, and the CIoU loss function was used for coordinate frame positioning in the prediction layer. Experiments on the improved YOLOv5m algorithm demonstrate that the mean recognition accuracy of the method improves by 10.7 percentage points over the original method while maintaining the original model size and detection speed, and better solves the problems of small scale, dense arrangement and mutual occlusion of targets in mask wearing detection tasks in crowded places. © 2023 IEEE.

FSA-Net: Rethinking the attention mechanisms in medical image segmentation from releasing global suppressed information.

Attention mechanism-based medical image segmentation methods have developed rapidly recently. For the attention mechanisms, it is crucial to accurately capture the distribution weights of the effective features contained in the data. To accomplish this task, most attention mechanisms prefer using the global squeezing approach. However, it will lead to a problem of over-focusing on the global most salient effective features of the region of interest, while suppressing the secondary salient ones. Making partial fine-grained features are abandoned directly. To address this issue, we propose to use a multiple-local perception method to aggregate global effective features, and design a fine-grained medical image segmentation network, named FSA-Net. This network consists of two key components: 1) the novel Separable Attention Mechanisms which replace global squeezing with local squeezing to release the suppressed secondary salient effective features. 2) a Multi-Attention Aggregator (MAA) which can fuse multi-level attention to efficiently aggregate task-relevant semantic information. We conduct extensive experimental evaluations on five publicly available medical image segmentation datasets: MoNuSeg, COVID-19-CT100, GlaS, CVC-ClinicDB, ISIC2018, and DRIVE datasets. Experimental results show that FSA-Net outperforms state-of-the-art methods in medical image segmentation.

Detection of Pneumonia Based on ResNet Improved by Attention Mechanism

In recent years, Covid-19 is one of the major health challenges facing the human population. Due to the highly infectious nature of Covid-19 and the difficulty of detecting symptoms in the early stages, it is definitely necessary to combine X-ray for the diagnosis of pneumonia. Using traditional neural networks such as VGG, ResNet, and DenseNet to diagnose pneumonia based on X-ray images faces a number of difficulties. These models have insufficient spatial information extraction capability and are prone to overfitting on the training set. The attention mechanism is a means to improve model performance by helping the model better extract channel and spatial features from the feature maps. To identify pneumonia more accurately, we combined the ResNet network and CBAM attention mechanism to design the ResNet101-cbam model with a series of data augmentation methods as well as training strategies. We used the same approach to add attention mechanisms to ResNet50, ResNet101 and ResNet152 and tested their performance. The results show that ResNet101-cbam is the best performing model overall. It achieved a recall of 0.8205, a precision of 0.822, and an accuracy of 0.8285 on the test set, while the original pretrained ResNet101 had a precision of 0.7280 and an accuracy of 0.7644. Its performance were better than the more complex model: ResNet152-cbam, a little bit, but the training speed is improved by more than 25%. More importantly, the model with the added attention mechanism effectively overcomes the effects of positive and negative sample imbalance. The ResNet101-cbam model can be used as a medical aid, which can improve diagnostic efficiency and help us better deal with large-scale pneumonia epidemics. © 2023 IEEE.

Fusing hybrid attentive network with self-supervised dual-channel heterogeneous graph for knowledge tracing

Recently the large-scale influence of Covid-19 promoted the fast development of intelligent tutoring systems (ITS). As a major task of ITS, Knowledge Tracing (KT) aims to capture a student's dynamic knowledge state based on his historical response sequences and provide personalized learning assistance to him. However, most existing KT methods have encountered the data sparsity problem. In real scenarios, an online tutoring system usually has an extensive collection of questions while each student can only interact with a limited number of questions. As a result, the records of some questions could be extremely sparse, which degrades the performance of traditional KT models. To resolve this issue, we propose a Dual-channel Heterogeneous Graph Network (DHGN) to learn informative representations of questions from students' records by capturing both the high-order heterogeneous and local relations. As the supervised learning manner applied in previous methods is incapable of exploiting unobserved relations between questions, we innovatively integrate a self-supervised framework into the KT task and employ contrastive learning via the two channels of DHGN, supplementing as an auxiliary task to improve the KT performance. Moreover, we adopt the attention mechanism, which has achieved impressive performance in natural language processing tasks, to effectively capture students' knowledge state. But the standard attention network is inapplicable to the KT task because the current knowledge state of a student usually shows strong dependency on his recently interactive questions, unlike the situation of language processing tasks, which focus more on the long-term dependency. To avoid the inefficiency of standard attention networks in the KT task, we further devise a novel Hybrid Attentive Network (HAN), which produces both the global attention and the hierarchical local attention to model the long-term and short-term intents, respectively. Then, by the gating network, a student's long-term and short-term intents are combined for efficient prediction. We conduct extensive experiments on several real-world datasets. Experimental results demonstrate that our proposed methods achieve significant performance improvement compared to existing state-of-the-art baselines, which validates the effectiveness of the proposed dual-channel heterogeneous graph framework and hybrid attentive network. © 2023 Elsevier Ltd

A New Pneumonia Detection Model Based on Transformer with Improved Self-Attention Mechanism

Clinical diagnosis based on computed tomography (CT) could be used, as part of diagnosis standard of COVID-19 pneumonia. Addressing the problem that accuracy of CT-based traditional pneumonia classification diagnosis models is relatively low when employed for classification of community-acquired pneumonia (CP), COVID-19 pneumonia (NCP) and normal cases, a new network model is proposed which combines application of Swin Transformer and multi-head axial self-attention (MASA) mechanism, to analyze CT images and make intelligence-assisted diagnosis. The method in detail is to partially replace traditional multi-head self-attention (MSA) mechanism in encoders of Swin Transformer by MASA. The improved model is applied to train and test on commonly used pneumonia CT dataset CC-CCII. The results show that the proposed network outperforms traditional networks ResNet50 and Vision Transformer in indicators of accuracy, sensitivity and F1-measure. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

A Predictive Model of COVID-19 mRNA Vaccine Reactivity based on Dual Attention for Federated Learning Scenarios

The base reactivity of the mRNA sequence has a significant impact on the effectiveness of the mRNA vaccine in fighting against the pandemic of COVID-19. The annotation of mRNA sequence reactivity value is a time-consuming and labor-intensive work, which belongs to the private digital assets of each medical institution. It is not practical to train a predictive model by pooling private data from various parties. Fortunately, federated learning techniques can serve to collaboratively train a predictive model among medical institutions while preserving respective digital assets. However, due to the scarcity of data from each participant, conventional sequential prediction mod-els often fail to perform well. To overcome such a challenge, we propose a reactivity value prediction model based on both the self-attention and the convolutional attention mechanisms only requiring a small dataset of labeled samples. Inspired by BERT, we first train a self-attention feature extraction model through self-supervision using both labeled and unlabeled mRNA samples. In this way, the information of mRNA in the semantic space is deeply mined. Then, a convolutional attention block follows the self-attention block, to extract the attention matrix from the base-pair probability matrix and adjacency matrix. By doing so, the attention matrix can compensate for the insensitivity of the self-attention mechanism to the spatial information of mRNA. By using the Open Vaccine RNA database, experiments show that our prediction model for unseen mRNA has a better performance than other state-of-the-art deep learning models that are used to process gene sequences. Further ablation experiments demonstrate that the existence of the dual attention mechanism reduces the risk of overfitting, resulting in an excellent generalization capability of our model. © 2022 IEEE.

Specific mining of COVID-19 reviews using deep learning methods

Sentiment classification is a valid measure to monitor public opinion on the COVID-19 epidemic. This study provides a significant basis for preventing the spread of adverse public opinion. Firstly, in epidemic texts, we use a convolutional neural network and bidirectional long short-term memory neural network BiLSTM model to classify and analyze the sentiment of the comment texts about the epidemic situation on Weibo. Secondly, embedded in the model layer to generate adversarial samples and extract semantics. Then, semantic information is weighted using the attention mechanism. Finally, the RMS optimizer is used to update the neural network weights iteratively. According to comparative experiments, the experimental results show that such four evaluation metrics as accuracy, precision, recall, and f1-score with our proposed model have obtained better classification performance. © 2022 IEEE.

Attention-Based Deep Learning Approach for Semantic Analysis of Chest X-Ray Images Modality

The world today is suffering from a huge pandemic COVID-19 that has infected 106M people around the globe causing 2.33M deaths, as of February 9, 2021. To control the disease from spreading more and to provide accurate healthcare to existing patients, detection of COVID-19 at an early stage is important. As per the World Health Organization, diagnosing pneumonia is a common way of detecting COVID-19. In many situations, a chest X-ray is used to determine the type of pneumonia. However, writing a report for every chest X-ray becomes a tedious and time-taking task for physicians. We propose a novel method of creating reports from chest X-rays images automatically via a deep learning model using image captioning with an attention mechanism employed through CNN–LSTM architecture. On comparing the model that does not use an attention mechanism with our approach, we found that accuracy was increased from 80% to 87.5%. In conclusion, we found that results generated with attention mechanism are better, and the report thus produced can be utilized by doctors and researchers worldwide to analyze new X-rays in lesser time. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

SMA-Net: Sobel Operator Combined with Multi-attention Networks for COVID-19 Lesion Segmentation

Coronavirus disease 2019 (COVID-19) has been spreading since late 2019, leading the world into a serious health crisis. To control the spread rate of infection, identifying patients accurately and quickly is the most crucial step. Computed tomography (CT) images of the chest are an important basis for diagnosing COVID-19. They also allow doctors to understand the details of the lung infection. However, manual segmentation of infected areas in CT images is time-consuming and laborious. With its excellent feature extraction capabilities, deep learning-based method has been widely used for automatic lesion segmentation of COVID-19 CT images. But, the segmentation accuracy of these methods is still limited. To effectively quantify the severity of lung infections, we propose a Sobel operator combined with Multi-Attention networks for COVID-19 lesion segmentation (SMA-Net). In our SMA-Net, an edge feature fusion module uses Sobel operator to add edge detail information to the input image. To guide the network to focus on key regions, the SMA-Net introduces a self-attentive channel attention mechanism and a spatial linear attention mechanism. In addition, Tversky loss function is adopted for the segmentation network for small size of lesions. Comparative experiments on COVID-19 public datasets show that the average Dice similarity coefficient (DSC) and joint intersection over Union (IOU) of proposed SMA-Net are 86.1% and 77.8%, respectively, which are better than most existing neural networks used for COVID-19 lesion segmentation. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

SPATIAL GRAPH ATTENTION AND CURIOSITY-DRIVEN POLICY FOR ANTIVIRAL DRUG DISCOVERY

We developed Distilled Graph Attention Policy Network (DGAPN), a reinforcement learning model to generate novel graph-structured chemical representations that optimize user-defined objectives by efficiently navigating a physically constrained domain. The framework is examined on the task of generating molecules that are designed to bind, noncovalently, to functional sites of SARS-CoV-2 proteins. We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure - this capability is of considerable interest in synthetic biology and drug discovery. An attentional policy network is introduced to learn the decision rules for a dynamic, fragment-based chemical environment, and state-of-the-art policy gradient techniques are employed to train the network with stability. Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation. In experiments, our framework achieved outstanding results compared to state-of-the-art algorithms, while reducing the complexity of paths to chemical synthesis. © 2022 ICLR 2022 - 10th International Conference on Learning Representationss. All rights reserved.

A New Classification Method for COVID-19 CT Images Based on Transfer Learning and Attention Mechanism

Nowadays, the typical tools employed in the diagnosis of the pandemic coronavirus disease, COVID-19, including Real-Time Reverse Transcription Polymerase Chain Reaction (RT-PCR) and Polymerase Chain Reaction (PCR), which are less sensitive, time-consuming, and demanding assistance from expert medical personnel assistance. Computed tomography (CT) as the artificial intelligence (AI) technological utilized in high accurate COVID-19 infection screening in a short amount of time is tremendously helpful. To address those limitations mentioned above, In this paper, a robust, optimized model for detection of the COVID-19 automatically in digital CT images is proposed utilizing the technique based on transfer learning and attention mechanism derived from deep learning. MobileNet-V1 architecture of transfer learning was applied to make the model more lightweight and reduce the computation, while setting to be the pre-trained mode meanwhile. In addition, the attention mechanism of SENet's called Squeeze-and-Excitation (SE) module was employed to let it learn the significance of various channel features automatically. Two experiments, with transfer learning and attention mechanism technique or not, were employed to assess the function of the model. Noteworthy, the accuracy, precision, recall, and F1-score were 95.97%, 93.47%, 94.27%, and 94.01% respectively. The results reveal that the optimized approach outperform the comparative models. © 2022 IEEE.

AM-DenseNet:A Novel DenseNet Framework using Attention Mechanisms for COVID-19 CT Image Classification

We present a novel DenseNet framework with attention mechanisms (AM-DenseNet) to extract lung feature of 1 COVID-19 patient. In AM-DenseNet, a lightweight Efficient Channel Attention (ECA) structure is added at the end of each dense connection to introduce an attention mechanism to discovery local lesion domain. We compare our AM-DenseNet to VGG-16, ResNet-50 and DenseNet-121 on CT image dataset of COVID-19 patients. According to the experimental results, we conclude that the classification performance of AM-DensNet framework can be significantly enhanced under the effect of attention mechanism. The AM-DensNet shows better classification performance than the compared models. © 2022 IEEE.

DCA-Net: A W-shaped network for segmenting COVID-19 infected area from medical images

Coronavirus disease (COVID-19) poses a significant threat to humans in 2019. Automated and accurate segmentation of the infected region of COVID-19 computed tomography (CT) images can help doctors diagnose and treat the disease. However, the variable shape of COVID-19 infected areas, which can be easily confused with other lung tissues, poses a challenge for CT image segmentation. To address this problem, a deep learning-based convolutional neural network is proposed for the automatic segmentation of COVID-19 lung infection regions. Our proposed segmentation method uses a U-Net network as the backbone, constructed as a coarse to fine segmentation network. Firstly, we introduce our designed contour-enhanced module (CA) in the coarse segmentation network to effectively extract the lung region;secondly, we introduce our designed multi-scale feature attention module (MFA) in the fine segmentation network to enable the network to extract spatial efficiently and channel information, better focus on quantifying the effective region, and enhance the network segmentation effect. Using the COVID-19 public dataset, the proposed network achieves the best segmentation results. The Dice, IOU, F1-Score, and Sensitivity metrics reach 88.74%, 78.73%, 86.58%, and 88.16%, respectively. DCA-Net can efficiently segment the COVID-19 infected region, which can be of great clinical benefit. © 2022 IEEE.

Short-Term State Electricity Load Forecasting Based on Transfer-Informer

The worldwide COVID-19 pandemic has caused an enormous impact on the operation mode of human society. Such sudden events bring sharp fluctuations and data inadequacy in datasets of several areas, which leads to challenges in solving related problems. Traditional deep learning models like CNN have shown relatively poor performance with small datasets during the COVID-19 pandemic. This is because the data insufficiency and fluctuations lead to serious problems in the training process. In our work, an Informer framework combined with Transfer learning methods (Transfer-Informer) is proposed to solve the data insufficiency in emergency situations, as well as to provide a more efficient self-attention mechanism for deep feature mining, with two distinctive advantages: (1) The ProbSpares self-attention mechanisms, which enables the proposed model to highlight dominant information and extract more typical features from time-series datasets. (2) The Transfer learning framework improves the generalization capability of the model, by transferring basic knowledge from normal situations to emergency cases with fewer data. In our experiments, Transfer-Informer is applied to short-term load forecasting, which achieves better predicting accuracy than traditional models. The empirical results indicate that the proposed model has put forward a baseline for short-term load forecasting in emergency situations and provided a feasible method to tackle sudden fluctuations in real problem-solving. © 2022 IEEE.

Short-Term State Electricity Load Forecasting Based on Transfer-Informer

The worldwide COVID-19 pandemic has caused an enormous impact on the operation mode of human society. Such sudden events bring sharp fluctuations and data inadequacy in datasets of several areas, which leads to challenges in solving related problems. Traditional deep learning models like CNN have shown relatively poor performance with small datasets during the COVID-19 pandemic. This is because the data insufficiency and fluctuations lead to serious problems in the training process. In our work, an Informer framework combined with Transfer learning methods (Transfer-Informer) is proposed to solve the data insufficiency in emergency situations, as well as to provide a more efficient self-attention mechanism for deep feature mining, with two distinctive advantages: (1) The ProbSpares self-attention mechanisms, which enables the proposed model to highlight dominant information and extract more typical features from time-series datasets. (2) The Transfer learning framework improves the generalization capability of the model, by transferring basic knowledge from normal situations to emergency cases with fewer data. In our experiments, Transfer-Informer is applied to short-term load forecasting, which achieves better predicting accuracy than traditional models. The empirical results indicate that the proposed model has put forward a baseline for short-term load forecasting in emergency situations and provided a feasible method to tackle sudden fluctuations in real problem-solving. © 2022 IEEE.

An Improved EfficientNet Model and its Applications in Pneumonia Image Classification

Since the outburst of COVID-19, the medical system has been facing great challenges due to the explosive growth in detection and treatment needs within a short period. To improve the working efficiency of doctors, an improved EfficientNet model of Convolutional Neural Network (CNN) was proposed and applied for the diagnosis of pneumonia cases and the classification of relevant images in the present study. First, the acquired images of pneumonia cases were divided to determine the zones with target features, and image size was limited to improve the training speed of the network. Meanwhile, reinforcement learning was performed to the input dataset to further improve the training effect of the model. Second, the preprocessed images were inputted into the improved EfficientNet-B4 model. The depth and width of the model, as well as the resolution of the input images, were determined by optimizing the combination coefficient. On this basis, the model was scaled, and then its ability in extracting the features of deep-layer images was strengthened by introducing the attention mechanism. Third, the learning rate was adjusted by using the Adaptive Momentum (ADAM), and the training efficiency of the model was accelerated. Finally, the test set was inputted into the trained model. Results demonstrate that the improved model could detect 98% of patients with pneumonia and 97% of patients without pneumonia. The accuracy rate, precision rate, and sensitivity of the model were generally improved. Lastly, the training and test results of VGGNet, SqueezeNet-Elus, SqueezeNet-Relu, and the improved EfficientNet-B4 models were compared and evaluated. The improved EfficientNet-B4 model achieved the highest comprehensive accuracy rate, reaching 92.95%. The proposed method provides some references to the application of the CNN model in image classification and recognition. © 2022 School of Science, IHU. All Rights Reserved.

MPSKT: Multi-head ProbSparse Self-Attention for Knowledge Tracing

Over the past two years, COVID-19 has led to a widespread rise in online education, and knowledge tracing has been used on various educational platforms. However, most existing knowledge tracing models still suffer from long-term dependence. To address this problem, we propose a Multi-head ProbSparse Self-Attention for Knowledge Tracing(MPSKT). Firstly, the temporal convolutional network is used to encode the position information of the input sequence. Then, the Multi-head ProbSparse Self-Attention in the encoder and decoder blocks is used to capture the relationship between the input sequences, and the convolution and pooling layers in the encoder block are used to shorten the length of the input sequence, which greatly reduces the time complexity of the model and better solves the problem of long-term dependence of the model. Finally, experimental results on three public online education datasets demonstrate the effectiveness of our proposed model. © 2022 Association for Computing Machinery.

Classification of German Jungian Extraversion and Introversion Texts with Assessment of Changes during the COVID-19 Pandemic

The corona pandemic and countermeasures such as social distancing and lockdowns have confronted individuals with new challenges for their mental health and well-being. It can be assumed that the Jungian psychology types of extraverts and introverts react differently to these challenges. We propose a Bi-LSTM model with an attention mechanism for classifying introversion and extraversion from German tweets, which is trained on hand-labeled data created by 335 participants. With this work, we provide this novel dataset for free use and validation. The proposed model achieves solid performance with F1 = .72. Furthermore, we created a feature engineered logistic model tree (LMT) trained on hand-labeled tweets, to which the data is also made available with this work. With this second model, German tweets before and during the pandemic have been investigated. Extraverts display more positive emotions, whilst introverts show more insight and higher rates of anxiety. Even though such a model can not replace proper psychological diagnostics, it can help shed light on linguistic markers and to help understand introversion and extraversion better for a variety of applications and investigations. © European Language Resources Association (ELRA)